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77ns  research  is  designed  to  identify  school  districts  in  Massachusetts  whose  student  test  scores 
exceed  the  scores  predicted  by  the  district's  demographic  characteristics.  The  work  is  not 
interuied  to  rank  districts'  performances  hut  rather  to  highlight  the  efforts  of  districts  whose 
students  are  exceeding  what  they  would  he  expected  to  achieve  on  statewide  standardized  tests. 
The  goal  is  to  enable  other  districts  to  study  and  learn  from  the  efforts  of  systems  identified  as 
effective  in  this  analysis. 
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"Education,  then,  beyond  all  other  devices  of  human  origin, 
is  the  great  equalizer  of  the  conditions  of  men..." 

—  Horace  Mann 


The  Effectiveness  Index:  Year  II 

The  first  analysis  of  school  district  effectiveness  came  out  in  February  of  1999  and 
evaluated  the  1998  MCAS  in  terms  of  district  demography.  The  central  tool  of  that 
analysis  was  the  Effectiveness  Index  methodology  that  examines  the  relationship 
between  selected  demographic  characteristics  and  educational  outcomes.  These 
characteristics  include:  average  education  level,  average  income,  poverty  rate,  single- 
parent  status,  language  spoken,  and  percentage  of  school-age  population  enrolled  in 
private  schools.  These  variables  were  chosen  because  they  correlate  with  achievement 
and  because  the  education  literature  identifies  them  as  connected  to  academic 
performance. 

Researchers  ranging  from  James  Coleman  in  the  1960s  to  James  Comer  in  the 
1990s  have  demonstrated  that  community  demographics  play  a  major  role  in  how  well 
children  do  in  school.  The  Effectiveness  Index  provides  a  means  of  isolating  the  role 
played  by  community  characteristics  concerning  student  performance  on  statewide 
educational  assessments.  With  a  community's  achievement  context  factored  into  its  test 
results,  it  is  possible  to  know  how  much  value  school  systems  add  to  demographic 
expectations. 

The  Effectiveness  Index  identifies  school  districts  that  add  value  to  the  learning 
readiness  of  their  students  as  indicated  by  higher-than-predicted  test  scores.  Identifying 
such  systems  is  a  first  step  to  determining  if  they  are  indeed  providing  more  effective 
educational  services  to  their  students.  Identifying  best  practices  in  effective  systems 
that  are  demographically  similar  to  less  effective  systems  may  help  those  systems 
improve  their  school  services. 
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This  second  edition  incorporates  several  new  elements  into  the  process: 

1)  More  districts  are  included.  Last  year's  analysis  was  confined  to  communities 
of  over  6,000  population.  This  year's  evaluation  considers  any  school  district  where  at 
least  50  students  took  the  MCAS  exam  in  a  grade.  Last  year's  work  covered  about  93% 
of  the  students  in  the  Commonwealth.  This  year's  extends  to  about  97%  of  the  student 
population  taking  MCAS. 

2)  Regional  school  districts  are  included  in  this  year's  work.  Last  year,  only 
commimities,  not  regional  districts  which  are  comprised  of  several  commimities,  were 
considered.  That  meant  that  individual  communities  were  evaluated  outside  of  their 
regional  school  identity.  For  example,  last  year  Pembroke  was  evaluated  as  Pembroke 
in  10th  grade  even  though  Pembroke  is  part  of  the  Silver  Lake  Regional  District  for 
high  school.  This  year's  analysis  looks  at  school  district  performance  on  a  regional 
district  basis  where  appropriate.  Commvmity  demographics  have  been  factored  to 
reflect  the  regional  school  district  characteristics. 

Observations 

•  We  see  many  repeat  performers.  Many  districts  that  outperformed  their 
demography  last  year  did  the  same  this  year.  These  include  Wobum,  Harvard,  North 
Reading,  and  Norwood.  This  is  not  surprising;  a  system  that  had  organized  itself  to 
enhance  student  achievement  in  1998  is  likely  to  have  kept  that  up  in  1999. 

•  Some  of  the  new  over-performers  made  the  list  this  year  because  they  were 
not  large  enough  to  have  been  considered  last  year.  Orleans,  a  strong  over-performer 
this  year,  was  too  small  to  be  included  in  last  year's  work.  With  smaller  districts 
included  this  year,  Orleans  is  in  the  mix. 

•  Districts  that  over-perform  their  demography  tend  to  be  middle-class  or 
demographically  advantaged  commimities.  Generally,  upper-demography 
commimities  are  two  to  three  times  more  likely  to  over-perform  than  commimities  that 
are  of  lower  demography.  This  is  unsettling  in  that  middle  and  upper-middle  class 
communities  do  not  need  to  over-perform  their  demography  to  meet  state 
achievement  standards.  Based  on  two  years  of  MCAS,  most  of  their  students  perform 
well  enough  now  to  pass  the  MCAS  graduation  requirements.  Districts  that  are 
disadvantaged  need  to  overcome  their  demography  in  order  to  lift  more  of  their 
students  into  success  on  MCAS,  but  so  far  these  districts  are  having  a  hard  time 
outperforming  their  community  characteristics. 
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•  Districts  that  over-performed  their  demography  did  so  without  any  apparent 
benefit  from  high  per-pupil  school  spending  or  high  levels  of  new  state  education 
reform  aid.  Generally,  over-performers  spent  at  or  below  state  average  and  were  not 
the  recipients  of  generous  amounts  of  Chapter  70  (education  assistance  and  reform)  aid. 
This  fact  is  interesting  in  that  providing  additional  funding  is  a  major  reform  tool  of  the 
Education  Reform  Act  of  1993.  Based  on  this  analysis,  that  extra  funding  has  not 
necessarily  purchased  the  systemic  change  that  enables  systems  to  help  their  students 
perform  above  their  demographic  expectations. 

•  So  far  after  seven  years  of  reform  funding,  there  is  little  evidence  that  the 
schools  have  changed  in  any  fundamental  ways.  MCAS  scores  were  relatively  flat 
from  1998  to  1999.  This  is  of  concern  because  EdiicationWeek  pointed  out  that  "Of  the 
states  in  the  early  stages  of  testing,  only  Massachusetts  failed  to  post  significant  gains  in 
the  second  year  of  its  new  assessment."  (David  J.  Hoff,  EducationWeek,  Jan  26, 2000;  p. 
12;  'Testing's  Ups  And  Downs  Predictable.")  With  flat  early  results,  the  challenge  of 
improving  resxilts,  especially  for  demographically  disadvantaged  systems,  is  daunting. 

The  Importance  of  Identifying  Over-performing  Systems 

Identifying  systems  that  over-perform  their  demography  is  important  in  that 
such  systems  may  have  valuable  lessons  to  offer  similar  systems  in  their  efforts  to 
boost  student  achievement.  Most  of  the  over-performers  were  not  among  the 
demographically  disadvantaged  systems.  This  means  that  when  we  identify  low 
demography /over-performing  systems,  we  need  to  study  them  carefully  and  see  if 
they  indeed  do  have  lessons  to  teach  their  peers  across  the  Commonwealth.  After  two 
years  of  MCAS,  we  know  that  many  of  otir  less  advantaged  districts  have  far  to  go  to 
meet  new  state  standards,  so  helping  them  move  ahead  is  critically  important  to 
ultimate  success. 

The  real  battle  in  education  reform  is  to  change  the  way  disadvantaged  systems 
teach  their  children.  Approximately  60%  of  students  in  very  disadvantaged  systems  are 
not  now  in  a  position  to  meet  state  graduation  standards.  Those  systems  are  home  to 
266,000  students,  31%  of  the  state's  student  population.  These  are  future  citizens  who 
are  struggling  to  learn  how  to  read,  write,  and  do  basic  math  at  a  level  sufficient  to 
meet  state  standards  and  to  succeed  in  life.  Identifying  peer  systems  that  have  figured 
out  how  to  help  their  students  over-perform  their  demography  is  essential  to  making 
education  reform  work  in  Massachusetts. 
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Testing  plays  an  important  role  in  most  of  the  contemporary  school  reform 
efforts  in  the  United  States.  The  Massachusetts  education  reform  effort  is  no  exception. 
Its  testing  vehicle  is  the  Massachusetts  Comprehensive  Assessment  System  or,  as  it  is 
commonly  known,  the  MCAS. 

The  MCAS  is  a  battery  of  tests  that  is  given  each  year  to  students  in  Grades  4,  S, 
and  10  in  each  school  district.  The  MCAS  is  aligned  with  a  series  of  curriculum 
frameworks  that  are  being  developed  by  the  state  Department  of  Education.  MCAS 
covers  such  academic  subjects  as  math,  science,  and  literacy  skills,  with  more  subjects  to 
be  added  later.  The  test  scores  are  broken  down  by  individual  student,  school,  and 
district  The  scores  for  individual  students  are  available  to  their  parents,  teachers, 
principals,  and  superintendents.  The  scores  for  entire  schools  and  districts  are  available 
to  the  public 

The  chief  objective  of  the  state's  education  reform  initiative  is  to  enable  public 
school  students  to  achieve  a  certain  level  of  knowledge  and  skill.  The  Massachusetts 
Department  of  Education  has  established  this  level  by  setting  out  what  students  are 
expected  to  learn  in  each  basic  subject.  School  districts  are  supposed  to  see  to  it  that 
their  students  learn  what  they  are  expected  to  leam.  The  purpose  of  the  MCAS  is  to 
gauge  periodically  how  students  are  doing  as  they  try  to  achieve  this  level  of 
knowledge  and  skill. 

With  the  MCAS,  the  state  has,  for  the  first  time  in  its  history,  an  evaluation 
mechanism  that  measvires  how  much  progress  students  are  making  towards  meeting 
established  goals.  At  the  same  time,  individual  schools  districts  are  urged  to  anticipate 
and  complement  the  MCAS  by  developing  their  own  parallel  methods  of  assessing  how 
their  students  are  doing.  Thus,  the  education  reform  effort  uses  assessment  as  a  way  to 
help  all  students  move  toward  a  high  level  of  academic  achievement. 

Just  as  this  overall  effort  views  higher  student  achievement  as  its  end,  it  views 
the  improvement  of  the  public  schools  as  its  chief  means  to  achieve  this  end.  What 
happens  in  school  is  by  no  means  the  only  or  even  the  leading  influence  on  how  pupils 
currently  perform  on  standardized  academic  tests.  However,  what  happens  in  school 
obviously  is  the  only  means  that  is  currently  within  the  control  of  the  schools 
themselves.  So  it  is  the  only  means  of  reform  that  is  at  the  disposal  of  the  education 
improvement  effort  as  it  now  exists. 
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Improving  Our  Schools 

The  more  the  test  scores  can  be  used  to  inform  decisions  about  how  to  alter  what 
happens  in  school,  the  better  the  chances  to  make  the  schools  more  effective  in  helping 
their  students  learn  more.  Properly  used,  the  results  can  pinpoint  which  approaches  to 
teaching  and  learning  are  working  and  which  are  not.  The  MCAS  also  includes  an  array 
of  diagnostic  tools  that  let  teachers  and  administrators  spot  areas  where  students 
perform  poorly,  so  that  the  staff  can  work  with  the  students  to  mend  the  weaknesses. 

Consequently,  the  essence  of  education  reform  in  Massachusetts  can  be 
summed  up  in  a  few  words:  Better  student  performance,  through  more  effective 
schools. 

However,  for  the  MCAS  to  fulfill  its  intended  role  in  the  current  education 
reform  effort,  there  at  least  two  important  conditions  that  have  to  be  met. 

FIRST,  the  tests,  and  other  assessments,  must  be  fair  and  accurate.  They  must 
measvire  what  children  have  learned,  rather  than  just  their  social  or  economic 
backgroimd.  They  must  not  be  biased  for,  or  against,  any  group  of  students. 

SECOND,  the  tests  must  be  used  to  make  the  public  schools  more  effective. 
Thus,  the  scores  should  drive  an  ongoing  analysis  of  what  makes  the  school  experience 
effective.  They  must  provide  teachers  with  a  critical  piece  of  information  about  the 
potential  learning  problems  and  possibilities  of  individual  students.  And  the 
information  must  be  used  as  a  basis  for  helping  all  students  to  do  better. 

To  meet  the  second  condition,  we  must  be  able  to  use  the  MCAS  scores  as  one 
tool  to  discern  the  effectiveness  of  ovir  schools.  We  must  be  able  to  establish  how 
effective  they  are  today,  and  to  track  the  rise  or  fall  of  their  effectiveness  in  the  future. 
Thus,  finding  ways  to  measure  school  effectiveness  is  essential  to  education  reform. 

Measuring  Effectiveness 

Student  academic  performance,  including  how  students  do  on  MCAS  tests,  is 
influenced  by  two  broad  sets  of  factors:  school  factors  and  non-school  factors.  The  first 
entail  what  happens  in  school,  and  thus  what  is  within  the  control  of  the  school  district 
itself.  The  second  entails  conditions  outside  the  schools,  such  as  the  demographic 
profile  of  the  students  and  the  commimity.  As  we  look  at  a  given  district's  average 
score  on  an  MCAS  test,  we  have  to  be  able  to  discern  how  much  of  the  score  is  tied  to 
school  factors,  and  how  much  of  the  score  is  explained  by  non-school  factors. 
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We  cannot  begin  to  zero  in  on  jtist  how  effective  the  school  district  itself  is  unless 
we  can  distingxiish  between  the  respective  influences  of  the  two  types  of  factors.  Only 
then  can  we  discern  how  effectively  the  district  itself  performs,  and  how  much  it 
contributes  to  its  students'  average  performance  on  the  MCAS. 

The  Effectiveness  Index  (EI)  provides  insight  into  this  distinction,  and 
consequently  provides  some  measure  of  the  school  district's  contribution  to  its  student 
performance.  Thus,  it  supplies  a  piece  of  crucial  insight  as  to  which  schools  are  more 
effective. 

For  a  given  district,  the  Effectiveness  Index  gauges  the  impact  that  school  factors 
have  on  the  average  MCAS  score.  The  greater  the  positive  impact  of  the  school  factors, 
the  higher  the  district's  Effectiveness  Index  will  be. 

The  Effectiveness  Index  is  calctilated  in  the  following  manner  For  a  given  district, 
the  six  demographic  factors  are  used  as  the  basis  for  projecting  a  likely  average  score 
on  the  MCAS.  The  demographically-likely  score  is  then  compared  to  the  average  score 
that  the  students  in  the  district  actually  received.  The  Effectiveness  Index  is  the  niunber 
that  represents  the  difference  between  the  likely  score  and  the  actual  score. 

If  the  number  is  negative  -  if  the  actual  score  is  lower  than  the  likely  score  -  then 
this  suggests  that  what  is  happening  in  the  schools  in  the  district  is  not  enabling  its 
students  to  perform  beyond  the  demographic  expectations  for  them.  If  the  nimiber  is  a 
positive  number  -  if  the  actual  score  is  higher  than  the  likely 

score  -  then  this  suggests  that  what  is  happening  in  the  schook  is  helping  the  district's 
students  to  surpass  the  demographic  expectations  for  them.  (For  a  fuller  accoimt  of  the 
development  of  the  Effectiveness  Index,  please  see  Appendix  B.) 
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How  well  do  the  school  design  and  the  curriculum  promote  learning  for  all? 
Are  teachers  top-notch  professionals  who  have  both  the  skills  and  commitment  to  teach 
all  students?  Are  professional  development  activities  rigorously  aligned  with  efforts  to 
increase  student  achievement?  Is  there  strong,  solid  leadership  in  the  school?  Are 
there  high  expectations  for  all?  Are  parents  full  partners  in  their  children's  education? 
Are  there  adequate  resources  to  do  the  job?  These  are  all  questions  about  school 
factors  J 

In  the  research  reported  in  this  paper,  non-school  factors  consist  largely  of  the 
overlapping  demographic  conditions  of  family  life  and  commimity  life.  This  study 
utilizes  six  such  conditions  in  a  given  school  district:  its  median  level  of  educational 
attainment;  its  median  income  level;  its  percentage  of  households  above  the  poverty 
line;  its  percentage  of  single-parent  families;  its  percentage  of  non-English-speaking 
households;  and  its  level  of  private  school  enrollment.  Statistical  analysis  shows  that 
these  factors  form  much  of  the  non-school  influence  on  how  the  state's  students  do  on 
such  standardized  tests  as  the  MCAS.^ 

As  we  all  know,  students  in  advantaged  districts  tend  to  get  higher  standardized 
test  scores  than  students  in  disadvantaged  districts.  Thus,  if  a  district's  students  get  a 
high  average  score  on  an  MCAS  or  other  standardized  tests,  the  test  score  by  itself  does 
not  tell  us  how  much  of  the  score  is  explained  by  school  factors  and  how  much  is 
explained  by  non-school  factors.  A  high  score  might  be  tied  more  to  advantaged 
demography  than  to  what  actually  happens  in  the  district's  schools.  The  score  by  itself  is 
not  a  sound  guide  to  how  effective  the  school  district  is. 


^Per  pupil  expenditure  [PPE]  is  a  school  factor,  but  our  measures  of  it  are  not  always  reliable.  There  is 
no  standard  accounting  procedure  for  establishing  PPE.  For  example,  some  systems  might  include 
teacher  retirement  costs,  capital  costs,  federal  funds,  and  long-term  disability  obligations  in  their  per- 
pupil  spending  figure.  Others  might  not.  Therefore,  comparisons  across  districts  are  difficult  to  make. 

Other  family  and  community  conditions  are  crucial  to  student  success,  but  are  hard  to  observe  and 
measure.  One  would  have  to  monitor  many  families  and  communities  dosely  over  time  to  discern  how 
family  and  community  behavior  affect  school  outcomes.  How  many  books  are  read  in  the  family?  How 
much  time  is  taken  up  by  TV-watching?  How  do  the  community's  adults  treat  children  other  than  their 
own?  Does  the  community  mentor  its  young  people?  It  is  hard  to  get  reliable  answers  to  such  questions. 
But  we  do  know  that  the  children  of  advantaged  families  and  communities  are  more  likely  on  average 
to  have  resources  and  support,  and  children  of  less  advantaged  situations  are  less  likely  to  have  such. 
Therefore,  we  use  gross  measures  of  such  support  as  a  proxy  for  answers  to  the  more  specific  questions 
that  are  so  hard  to  pursue. 
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development  of  the  Effectiveness  Index,  please  see  Appendix  B.) 


9 


r 


What  the  Effectiveness  Index  Tells  Us:  Statewide  Results 

This  work  applied  the  Effectiveness  Index  methodology  to  the  1999  MCAS  scores 
of  school  districts  in  the  state.  A  few  districts  were  too  small  for  results  to  be  statistically 
reliable  so  they  are  not  included.  The  districts  considered  for  this  report  comprise  97% 
percent  of  the  total  student  population  of  those  taking  the  MCAS  in  1999. 

One  of  the  consistent  findings  of  this  research  is  that  demography  explains  most 
of  the  variation  in  test  scores  from  district  to  district.  Results  from  this  year's  research 
are  similar  to  results  from  last  year's  work:  about  84%  of  the  variation  in  test  results 
(scores  for  all  of  the  test-taking  students  for  the  nine  MCAS  tests  combined)  is  explained 
by  demography.  That  is  why  Weston  and  Way  land  have  high  MCAS  scores  and  why 
Holyoke  and  Brockton  have  low  MCAS  scores.  Thus,  though  demography  is  not 
destiny,  it  sets  a  strong  tendency. 

A  simple  way  to  depict  the  respective  contributions  that  demography  and  the 
schools  make  to  the  average  level  of  student  performance  on  the  MCAS  is  this: 

DEMOGRAPHY  +  school  =  average  score 

Despite  the  heavy  weight  of  demography  in  today's  learning  equation,  a 
nimiber  of  districts  achieved  test  scores  that  are  significantly  higher  than  their 
demography  predicts. 

Effective  and  Noteworthy  School  Districts 

The  Effectiveness  Index  lets  us  identify  two  types  of  school  districts  that  are 
interesting  in  terms  of  education  reform:  Effective  Districts  and  Noteworthy  Districts. 

An  EFFECTIVE  district  meets  two  specifications: 

1)  Its  Effectiveness  Index  is  a  positive  number  -  that  is,  its  actual 
score  on  the  test  is  higher  than  its  demographically  likely  score. 

2)  Its  actual  score  is  equal  to  or  higher  than  the  average  MCAS 
score  for  the  state  as  a  whole. 
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Thus,  a  district  that  meets  both  of  these  specifications  invites  further  and  closer 
scrutiny  to  determine  whether  its  practices  provide  a  worthwhile  model  for  other 
districts.  Not  all  districts  that  meet  the  two  effectiveness  specifications  will  prove  to 
have  lessons  to  teach  other  systems. 

Stoneham,  Norwood,  and  Millbury  are  examples  of  effective  school  districts. 
Actual  total  MCAS  scores  in  those  districts  are  substantially  higher  than  their 
demographically  likely  score.  Stoneham's  MCAS  scores  are  58  ranks  higher  than  its 
demography;  Norwood's  scores  are  42  ranks  higher;  and  Millbury 's  scores  are  34  ranks 
above  its  demographic  position.  Additionally,  each  of  these  districts'  total  1999  MCAS 
score  is  higher  than  the  statewide  average  score.^ 

A  NOTEWORTHY  district  fits  the  first  specification  but  doesn't  fit  the  second. 
Since  its  performance  helps  its  students  to  go  beyond  their  demography,  it  is  still 
worthy  of  note.  What  such  a  district  is  doing  educationally  can  hold  useful  lessons  for 
districts  that  are  demographically  similar,  but  do  not  outscore  their  demography.  And 
such  a  district  is  more  likely  to  deliver  a  return  on  future  public  investment  than  an 
ineffective  district. 

Here,  Everett  and  Gardner  are  outstanding  examples. 

•  For  the  second  year  in  a  row,  Everett's  overall  score  on  all  nine  tests  combined 
is  much  higher  than  its  demography  predicts.  Further,  Everett's  10th  grade  scores  were 
substantially  better  than  demography  would  suggest.  This  is  interesting  because  urban 
high  schools  so  far  have  proven  to  be  very  problematic  in  terms  of  implementing 
effective  reform. 

•  Gardner's  total  MCAS  scores  on  all  three  MCAS  tests  surpass  its  demographic 
prediction-  What  is  especially  encouraging  is  that  Gardner  middle  and  high  school 
students  greatly  over-performed  their  demography,  again  an  unusual  circumstance  for 
demographically  challenged  communities.^ 


'In  terms  of  demography  of  the  204  school  systems  whose  students  took  all  MCAS  tests,  Stoneham  ranks 
#109,  squarely  in  the  middle.  Its  combined  MCAS  scores  were  the  51st  highest  in  the  state.  Norwood 
ranks  #89  demographically;  its  combined  MCAS  scores  were  the  47th  highest  in  the  state.  Millbury 
ranks  123rd  demographically  and  had  the  89th  highest  MCAS  scores. 

*In  terms  of  demography  of  the  204  school  systems  whose  students  took  all  MCAS  tests,  Everett  ranks 
#186.  Its  combined  MCAS  scores  were  the  166nd  highest  in  the  state.  Gardner  ranks  #181st 
demographically;  its  combined  MCAS  scores  were  the  151st  highest  in  the  state. 
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Making  the  Grade 

MCAS  and  the  Class  of  2003  For  the  Qass  of  2003,  success  on  the  MCAS  is  currently 
defined  as  scoring  Needs  Improvement  or  better  on  both  the  Math  and  English 
Language  Arts  (ELA)  tests.  Although  students  take  the  Science  and  Technology  MCAS 
tests,  members  of  the  Qass  of  2003  will  not  have  to  pass  it  to  meet  state  requirements. 

Let  us  imagine  that  the  scores  obtained  on  last  year's  8th  grade  MCAS  by 
students  in  the  Qass  of  2003  cotmted  for  graduation.  For  students  who  attend  school  in 
the  25  most  advantaged  communities^,  3%  of  the  students  would  have  failed  the  ELA 
test  and  12%  would  have  failed  the  Math.  Since  students  must  pass  both  to  graduate, 
12%  of  students  in  demographically  advantaged  communities  would  have  failed. 

For  students  who  attend  school  in  the  25  most  disadvantaged  communities^, 
27%  of  the  students  would  have  failed  the  ELA  test  and  65%  would  have  failed  the 
Math.  Since  students  must  pass  both  to  graduate,  65%  of  students  in  demographically 
disadvantaged  communities  would  have  failed.  Children  in  the  schools  in  the  25  most 
demographically  challenged  communities  accotmt  for  31%  of  all  of  the  public  school 
students  in  Massachusetts. 

The  chart  on  the  following  page  illustrates  the  range  of  achievement  between 
advantaged  and  disadvantaged  commimities  in  Massachusetts.  This  presentation  looks 
at  the  state  in  terms  of  demographic  kind  of  community  (KOC)  with  cities  and  towns 
being  placed  in  one  of  six  KOC  dusters.  The  methodology  for  measuring  demography 
was  developed  in  a  dissertation,  (See  Appendix  B.) 


*rhe  25  most  advantaged  8th  grade  districts  demographically,  in  order  from  most  advantaged  down, 
are:  Weston,  Carlisle,  Dover-Sherbom,  Sudbury,  Wellesley,  Wayland,  Cohasset,  Concord,  Acton- 
Boxborough,  Medfield,  Winchester,  Duxbury,  Lincoln.  Lexington,  Southborough,  Longmeadow,  Norwell, 
Andover,  Holliston,  Westwood,  Westford,  Bedford,  Needham,  Hingham,  and  Hopkinton. 

^The  25  most  disadvantaged  8th  grade  districts  demographically,  in  order  from  most  disadvantaged 
up,  are:  Lawrence,  Chelsea,  Holyoke,  New  Bedford,  Lowell,  Springfield,  Fall  River,  Boston,  Lynn, 
Fitchburg,  Southbridge,  Worcester,  Revere,  Brockton,  Somerville,  Chicopee,  Taunton,  Everett,  Ludlow, 
Ware,  North  Adams,  Webster,  Gardner,  Leominster,  and  Maiden. 
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MCAS  GR  8:  Class  of  2003  Fail  Percentage 

70  -,  


Very  Advantaged      Upper  Mid-        Lower  Mid-        Challenged  Very 

Advantaged  Mass  Mass  Challenged 


Not  surprisingly,  a  district's  success  in  placing  its  students  in  the  pass  category  is  very 
dependent  upon  its  demography.  The  dimension  of  the  challenge  facing  education 
reformers  is  clear  as  we  end  the  first  phase  of  the  school  improvement  effort:  Find 
ways  to  boost  achievement  of  students  in  our  less  advantaged  communities. 

Note: 

Very  Advantaged  communities  include  towns  such  as  Wellesley,  Lexington,  Medfield,  and  Andover. 

Advantaged  communities  include  Franklin,  Marblehead,  Natick,  and  Newton. 

Upper  Mid-Mass  communities  include  Walpole,  Ipswich,  Brookline,  and  Chatham. 

Lower  Mid-Mass  communities  include  Beverly,  East  Bridgewater,  Stoneham,  and  Agawam. 

Challenged  communities  include  Dartmouth,  Quincy,  Peabody,  and  Methuen. 

Very  Challenged  communities  include  Everett,  Ware,  Fall  River,  and  Boston. 
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The  Importance  of  Reading  and  Writing 

While  students  need  to  develop  a  broad  range  of  skills  and  competencies, 
reading  and  writing  are  the  foundations  for  academic  success.  E.  D.  Hirsclv  Jr.,  author 
of  The  Schools  We  Need  and  the  developer  of  the  Core  Knowledge  Curriculum,  observes 
that "...  good  competency  tests  in  reading  turn  out  to  be  powerfully  indicative  of 
achieved  abilities  that  go  far  beyond  reading."  (E.  D.  Hirsch,  Education  Week,  Feb  2, 
2000;  p,  40,  in  "The  Tests  We  Need.")  Veteran  educators  understand  that  reading  is  the 
linchpin  of  academic  success. 

Having  solid  reading  and  writing  skills  are  necessary  conditions  for  doing  well 
on  the  MCAS  tests.  This  is  true  even  of  the  tests  in  Mathematics.  Many  of  the  problems 
on  the  Math  tests,  particularly  in  grade  eight  and  ten,  are  word  problems.  You  cannot 
tmderstand  these  problems  if  you  cannot  understand  the  words.  In  all  subjects, 
moreover,  many  questions  call  for  a  written  answer,  as  short  as  a  sentence  or  two  or  as 
long  as  an  essay  of  several  paragraphs. 

The  tables  in  Appendix  A  list  effective  districts  in  terms  of  student  performance 
in  various  subjects  including  reading.  Several  interesting  systems  that  are  effective  in 
teaching  reading  (ELA)  in  Grade  4  are  Orange,  Monson,  and  Stoughton.  Each  of  these 
districts  had  MCAS  scores  substantially  above  their  predicted  value.  In  Grade  8, 
Braintree,  Quabbin  Regional,  and  Northampton  solidly  over-scored  their  demography 
in  ELA. 

This  study  highlights  these  districts  because  they  might  have  lessons  to  offer 
to  other  districts  to  help  them  to  enhance  their  contribution  to  their  student's  future 
performance  of  the  MCAS.  It  is  especially  important  to  note  and  study  the  success  of 
demographically  challenged  and  demographically  average  commimities  in  exceeding 
expectations.  In  particular.  Orange,  a  very  challenged  system,  deserves  consideration 
for  adding  value  to  the  reading  skills  of  its  students. 
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\fiddle  Massachusetts 


In  the  demographic  ranking  of  Massachusetts  school  districts,  about  140  districts 
are  concentrated  in  the  demographic  middle  of  the  state/  These  districts,  with  over 
two-and-a-half  million  people,  make  up  what  might  be  called  Middle  Massachusetts 
(MidMass).  These  systems  may  be  well  suited  to  play  a  crucial  role  in  the  short-term 
future  of  education  reform. 

For  the  state  as  a  whole,  as  we  have  seen,  demographic  differences  among  the 
districts  explain  84%  of  the  variation  in  the  districts'  average  overall  test  scores.  All  or 
much  of  the  other  16%  of  the  variation  is  probably  explained  by  the  differences  in  how 
the  school  districts  themselves  behave. 

When  looking  at  either  end  of  the  demographic  ladder,  we  notice  that  scores 
tend  to  be  very  high  or  very  low.  While  there  is  some  variation,  the  solid  suburbs  score 
well.  Conversely,  the  cities  have  low  scores,  again  with  some  variation.  In  short,  the 
overwhelming  majority  of  students  in  advantaged  districts  pass  MCAS  easily;  most 
students  in  disadvantaged  districts  are  very  far  away  from  passing. 

The  pattern  in  Middle  Massachusetts  is  different.  Its  districts  exhibit  a  wide  range 
of  test  scores  ~  although  their  demography  is  relatively  similar. 

This  variation  can  be  seen  in  the  bar  graph  on  the  next  page  that  shows  total 
MCAS  scores  for  the  1999  MCAS  Grade  8  tests.  (This  utilizes  the  Grade  8  tests  because 
these  students  are  members  of  the  first  class  that  will  have  to  pass  MCAS  in  order  to 
qtialify  for  a  high  school  diploma.  Results  for  other  grade  levels  are  similar.) 


Because  of  regional  systems,  the  actual  number  of  Massachusetts  school  districts  varies  with  grade 
level.  There  are  243  Grade  4  districts,  231  Grade  8  districts,  and  223  Grade  10  districts. 
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For  each  of  the  25  districts  in  the  demographic  middle  of  the  state  -  the  districts 
that  form  the  middle  of  Middle  Massachusetts  itself  -  the  tip  of  the  bar  represents  its 
actual  test  score  on  the  1999  Grade  8  MCAS.  The  centerline  represents  the  score  each 
commimity  would  be  expected  to  achieve  based  on  its  demographics.  The  numbers  on 
the  bottom  of  the  chart  represent  MCAS  scaled  score  numbers.  Please  Note:  The 
communities  are  listed  to  the  left  of  their  MCAS  performance;  only  over-performers  are 
identified. 


MidMass:  TOTAL  MCAS  GR  8 


4 

CENTERLINE 


As  you  can  see,  there  is  wide  variation  in  actual  test  scores  in  towns  that  are  very 
similar  demographically.  Since  the  demographic  variation  is  slight,  but  the  variation  in 
test  scores  is  great,  this  pattern  suggests  that  much  of  the  variation  is  explained  less  by 
demography  than  by  differences  in  what  the  schools  of  MidMass  are  doing. 
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This  variation  has  practical  implications.  Wobum  is  a  district  that  over-performs 
its  demography.  Marlboro  is  a  demographically  similar  system  that  is  literally  right 
next  to  Wob\im  in  the  demographic  methodology.  For  Marlborough,  66%  of  the 
students  passed  the  Grade  8  Math  MCAS;  for  Wobum  the  pass  rate  was  82%.  (Passing 
is  defined  as  Needs  Improvement  or  better  in  Math  and  ELA.)  It  is  also  interesting  to 
note  that  Wobum's  per-pupil  expenditure  was  $5991;  Marlborough's  was  $6534. 
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1999  MCAS  Percent  Pass  Rate  GR  8 
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Further,  the  test  scores  of  MidMass  districts  with  high  positive  numbers  on  the 
Effectiveness  Index  are  just  as  high  as  the  scores  of  many  of  the  advantaged  districts. 
For  example,  for  Grade  4,  a  class  that  started  school  as  the  Education  Reform  Act  of 
1993  was  enacted,  the  total  MCAS  Grade  4  scores  of  Norwood,  a  MidMass  district,  are 
equal  to  the  scores  of  Westwood,  a  very  advantaged  community.  Similarly,  fourth 
graders  in  Pembroke,  another  MidMass  district,  scored  the  same  as  students  in 
Hamilton-Wenham,  an  advantaged  district.  It  is  important  to  note  that  both  Westwood 
and  Hamilton-Wenham  students  performed  very  well  on  the  MCAS;  the  story  here  is 
how  well  youngsters  in  Norwood  and  Pembroke  did. 
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So,  if  more  MidMass  districts  become  as  effective  as  Norwood,  Pembroke,  and 
Wobunv  then  more  MidMass  districts  will  get  test  scores  as  high  as  the  test  scores  of 
the  advantaged  districts. 

Moreover,  insofar  as  MidMass  districts  are  demographically  similar,  what 
makes  for  effective  schools  in  an  effective  districts  in  MidMass  is  more  likely  to  make 
for  effective  schools  in  an  ineffective  district  in  MidMass. 

Thus,  in  the  short  run,  MidMass  can  be  an  especially  fruitful  place  to  seek,  and 
expect  to  find,  a  relatively  swift  rise  in  MCAS  test  scores. 

Education  Reform  in  Massachusetts 

The  Education  Reform  Act  of  1993  provides  an  opportimity  to  transform  our 
schools.  The  Act  can  be  imderstood  in  terms  of  two  basic  changes  it  brought  about: 

1)  Sharply  increasing  the  amoimt  of  state  aid  targeted  at  disadvantaged,  low- 
spending  commtmities;  and 

2)  The  establishment  of  statewide  academic  performance  standards  (curriculum 
frameworks)  and  an  assessment  device  to  measure  progress  towards  meeting  those 
standards  (the  MCAS). 

The  MCAS  can  be  the  backbone  of  our  effort  to  make  the  schools  work  for  all 
students,  regardless  of  where  they  happened  to  be  bom  and  raised.  It  can  assess  the 
performance  of  districts,  schools,  and  individual  students,  and  it  can  inform  the  public 
about  its  schools.  More  importantly,  MCAS's  built-in  diagnostics  can  help  teachers  to 
help  all  children  learn  better.  Under  the  act,  increased  state  funds  have  provided 
substantial  amounts  of  new  money  for  many  districts  to  use  for  reform. 

End  Note 

This  study  captures  the  role  that  demography  plays  in  student  performance  on 
the  1999  MCAS.  While  demography  is  not  destiny,  it  does  establish  a  tendency.  If  we 
overlook  the  tendency  of  disadvantaged  districts  to  produce  low  scores  and  under- 
educated  students,  then  we  will  continue  to  consign  the  children  of  those  districts  to  a 
future  of  imfulfilled  potential.  After  seven  years  of  education  reform  in  Massachusetts, 
demography  still  plays  too  large  a  role  in  the  school  performance  of  our  children. 
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Without  a  broad  long-term  effort  to  do  what  is  needed  to  enhance  performance 
by  all,  especially  students  in  our  disadvantaged  systems,  we  can  expect  more  of  the 
same:  A  polarization  of  academic  performance  that  troubles  even  those  whose  children 
are  fortunate  enough  to  have  been  bom  into  a  situation  that  makes  a  powerful 
contribution  to  their  academic  success. 

After  several  years  of  education  reform,  not  everything  is  dear.  What  is 
indisputable,  however,  is  that  we  know  that  children  in  advantaged  school  districts  are 
equipped  to  succeed  in  meeting  state  MCAS  graduation  requirements.  We  also  know 
that  children  from  disadvantaged  districts,  after  billions  of  dollars  in  new  spending,  are 
still  woefully  under-prepared  for  success.  "All  children  can  learn"  certainly  sounds  nice; 
the  evidence  is  dear  that  we  are  far  away  from  developing  the  kinds  of  effective  schools 
where  all  children  in  fact  do  learn. 

Massachusetts  stands  at  a  critical  crossroads.  The  elements  are  in  place  for 
exdting  statewide  reform,  but  the  barriers  to  change  are  substantial.  The  Education 
Reform  Act  of  1993  is  being  considered  for  reauthorization  in  the  spring  of  2000.  While 
much  has  been  accomplished,  much  remains  to  be  done. 


19 


Appendices 


Appendix  A:  Listings  of  Effective  School  Districts 
Appendix  B:  Deriving  the  Effectiveness  Index 


r 


APPENDIX  A: 

Effectiveness  Indexes  on  the  1999  MCAS 


The  following  charts  list  the  top  sixty  districts  in  terms  of  their  score  on  the 
Effectiveness  Index.  The  Effectiveness  Index  is  the  difference  between  the  likely  score 
based  on  a  district's  demography  and  the  district's  actual  score  on  the  MCAS.  Districts 
are  listed  from  higher  to  lower  Effectiveness  Indexes. 

When  analyzing  anything  about  student  performance,  it  is  important  to  understand 
that  many  Massachxisetts  commimities  are  part  of  regional  school  districts  and  that, 
depending  on  grade  level,  a  commxmity's  school  district  may  change.  Thus,  the  Grade  4 
results  reflect  evaluating  a  different  set  of  school  districts  than  do  the  Grade  8  results 
and  do  the  Grade  10  results.  The  following  charts  represent  97%  of  the  state's  students. 
Due  to  small  sample  size,  those  districts  where  fewer  than  50  students  took  the  MCAS 
are  not  included. 

There  are  351  communities  in  Massachusetts.  There  are  204  school  districts  whose 
children  took  MCAS  tests  in  all  three  grades.  The  other  communities  are  part  of 
various  regional  school  districts.  Thus,  the  only  way  to  compare  total  MCAS  score  is  to 
limit  that  analysis  to  the  204  systems  whose  children  in  fact  took  MCAS  tests  at  all  three 
grade  levels.  Grades  4, 8,  and  10. 

Listings  are  provided  for  the  following  categories: 

1)  School  Districts  (including  Regional  Districts)  for  each  grade  level; 

Grade  fovir  performance:  all  three  subjects  combined  (English  Language  Arts,  Math, 
Science) 

Grade  four  performance:  English  Language  Arts 
Grade  four  performance:  Math 
Grade  four  performance:  Science 

Grade  eight  performance:  All  three  subjects  combined  (English  Language  Arts,  Math, 
Science.  (NOTE:  The  History  test  results  are  not  included  in  this  analysis.) 
Grade  eight  performance:  English  Language  Arts 
Grade  eight  performance:  Math 
Grade  eight  performance:  Science 
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Grade  ten  performance:  All  three  subjects  combined  (English  Language  Arts,  Math, 
Science) 

Grade  ten  performance:  English  Language  Arts 
Grade  ten  performance:  Math 
Grade  ten  performance:  Science 

2)  School  Districts  whose  students  took  all  MCAS  exams  in  all  three  grades 
in-district 

This  is  also  a  listing  of  the  60  most  Effective  Districts  of  the  204  systems  whose  students 
participated  in  all  three  grades  of  MCAS  in-district.  This  listing  is  derived  from  the  same 
basic  methodology  as  are  the  lists  for  Effective  Ehstricts  in  individual  grades  and 
subjects.  It  is  based  on  looking  at  the  difference  between  a  district's  demographic  rank 
and  its  MCAS  score  rank.  For  example,  if  a  district  was  ranked  the  100th  highest 
demographically  and  its  students  achieved  a  combined  MCAS  score  that  was  the  80th 
highest  in  the  state,  that  district  would  be  a  +20  in  terms  of  overall  effectiveness.  If  that 
district  that  was  ranked  #100  demographically  and  achieved  a  combined  score  that  was 
the  120th  highest,  it  would  be  a  -20  in  terms  of  overall  effectiveness. 


The  following  School  District  Effectiveness  Listings  are  for  each  grade  level  and 
subject  and  for  total  MCAS  score.  The  districts  with  the  highest  Effectiveness 
Indexes  (over-performance  of  their  demography)  are  to  the  upper  left  of  the  page;  the 
districts  to  the  lower  right  have  lower  Effectiveness  Indexes  but  still  over-performed. 
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Total  4  MCAS  1999  -  EFFECTIVE  DISTRICTS 


Orleans 

Woburn 

Harvard 

Orange 

Pembroke 

Marion 

Norwood 

Granby 

Eastham 

North  Reading 

Plymouth 

Sharon 

Newton 

Arlington 

East  Long  meadow 

Lexington 

Topsfield 

Everett 

Shrewsbury 

Monson 

Sandwich 

Chelsea 

Braintree 

Ayer 

Acton 

Wachusett  Reg. 
Stoughton 
Beverly 
Boylston 

West  Bridgewater 


MOST  EFFECTIVE 
▲ 


Scituate 
Franklin 
Lenox 
Nashoba 

Pentucket  Regional 

North  Attleborough 

Auburn 

Belmont 

Westford 

Ludlow 

Brewster 

Dartmouth 

Clinton 

Wrentham 

Dracut 

Hudson 

West  Boylston 

Winchester 

Westbo  rough 

Boxborough 

Dan  vers 

Falmouth 

Brookline 

Littleton 

Harwich 

Sturbridge 

Hampden  Wilbraham 

Gateway 

Canton 

Quabbin 


Arrow  direction  Indicates  more  effective 
Noteworthy  Districts  in  italics 


ELA  4  MCAS  1999  -  EFFECTIVE  DISTRICTS 


Pembroke 

Woburn 

Orleans 

Eastham 

Orange 

East  Longnfieadow 

Lenox 

Everett 

Harvard 

North  Reading 

Acton 

Monson 

Stoughton 

Sharon 

Longmeadow 

Arlington 

Newton 

Lawrence 

Scituate 

Braintree 

West  Boylston 

Wrentham 

Marion 

Danvers 

Norwood 

Lexington 

Topsfield 

Sandwich 

Pentucket  Regional 
Franklin 


MOST  EFFECTIVE 


Boylston 

Wachusett  Reg. 

Waipole 

Beverly 

Mansfield 

Westborough 

Plymouth 

Uxbridge 

Auburn 

Wellesley 

Weymouth 

Berkshire  Hills 

Falmouth 

Kingston 

Dartmouth 

Canton 

Nashoba 

Marshfield 

South  borough 

North  Attleborough 

Ayer 

Brookline 
Lee 

Hudson 

Wilmington 

Acushnet 

Shrewsbury 

Medfield 

Easton 

Harwich 


EFFECTIVE 


Arrow  direction  Indicates  more  effective 
Noteworthy  Districts  in  italics 


Math  4  MCAS  1999  -  EFFECTIVE  DISTRICTS 


Orleans 

Orange 

Granby 

Harvard 

Woburn 

Marion 

Newton 

Shrewsbury 

Plymouth 

Chelsea 

North  Reading 

Lexington 

Pembroke 

Clinton 

Wachusett  Reg. 

Winchester 

Ayer 

Norwood 

Belmont 

Beverly 

Sandwich 

Sturbridge 

Sharon 

Arlington 

Dracut 

Everett 

Reading 

Nashoba 

Groton  Dunstable 
Triton 


MOST  EFFECTIVE 
▲ 


Franklin 

Monson 

Braintree 

Topsfield 

West  Bridgewater 

Harwich 

East  Longmeadow 

Ludlow 

Sutton 

Webster 

Dartmouth 

Westford 

Carlisle 

Westborough 

Littleton 

Boxborough 

Auburn 

Wrentham 

Acton 

Falmouth 

Scituate 

Hingham 

Leominster 

Canton 

Methuen 

Chelmsford 

Gateway 

Brewster 

Foxborough 

Danvers 


EFFECTIVE 


Arrow  direction  Indicates  more  effective 
Noteworthy  Districts  in  italics 
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Science  4  MCAS  1999  -  EFFECTIVE  DISTRICTS 


Orleans 

Eastham 

Harvard 

Norwood 

Woburn 

Marion 

Granby 

Pembroke 

Sharon 

Plymouth 

Topsfield 

Orange 

Arlington 

Quabbin 

North  Reading 

North  Attleborough 

East  Long  meadow 

Boylston 

Hampden  Wilbraham 

Newton 

Brookline 

Wilmington 

Chelsea 

Hudson 

Braintree 

West  Bridgewater 

Everett 

Monson 

Brewster 

Acton 


MOST  EFFECTIVE 


Lexington 

Dedham  i 
Stoughton 
Carver 
Shrewsbury 
Pentucket  Regional 
Westford 
Ayer 

Sandwich 
Ludlow 
Somerset 
Scituate 
Fall  River 
Auburn 
Longmeadow 
Mohawk  Trail 
Needham 
Pea body 
Williamstown 
Ashubrnham  Westminster 
Gateway 
Lenox 
Lee 
Oxford 
Boxborough 
Nashoba 
West  Boylston 
Dartmouth 
Swampscott  EFFECTIVE 
Franklin 


Arrow  direction  Indicates  more  effective 
Noteworthy  Districts  in  italics 


Total  8  MCAS  1999  -  EFFECTIVE  DISTRICTS 


Harvard 

Stoneham 

Tyngs  borough 

Westborough 

North  Reading 

Hadley 

Lenox 

Sandwich 

Amherst  Pelham 

Quabbin 

Newbury  port 

Southern  Berkshire 

Clinton 

Bedford 

Medway 

Berlin  Boylston 

Pentucket  Regional 

Northampton 

Lawrence 

Gardner 

Berkshire  Hills 

Wayland 

West  Bridgewater 

Chelsea 

Belchertown 

Norwood 

Swampscott 

Braintree 

Woburn 

Peabody 


MOST  EFFECTIVE 


Triton 

Wachusett  Reg. 

West  Tisbury 

Brookline 

Manchester 

Masconomet 

Foxborough 

King  Philip 

Stoughton 

Tantasqua 

Auburn 

Abington 

Ludlow 

Easthampton 

Dartmouth 

Arlington 

North  Brookfield 

Lexington 

East  Longmeadow 

Newton 

Medfield 

Uxbridge 

Carver 

Leicester 

Holbrook 

Frontier 

Ware 

Sutton 

Westford 

Agawam 


EFFECTIVE 


Arrow  direction  Indicates  more  effective 
Noteworthy  Districts  in  italics 


ELA  8  MCAS  1999  -  EFFECTIVE  DISTRICTS 


Tyngs  borough 
Stoneham 
Braintree 
Quabbin 

East  Longmeadow 

Woburn 

Harvard 

Sandwich 

Pentucket  Regional 

Bedford 

Lenox 

Northampton 

Lawrence 

North  Reading 

Gardner 

Clinton 

Swampscott 

Southern  Berkshire 

Westborough 

Berkshire  Hills 

Berkley 

Uxbridge 

Brookline 

Norton 

Hadley 

Arlington 

Norwood 

Manchester 

Frontier 

Medfield 


MOST  EFFECTIVE 


Wakefield 

Dartmouth 

Amherst  Pelham 

Abington 

Medway 

Longmeadow 

Berlin  Boylston 

Central  Berkshire 

Ralph  Mahar 

Chelsea 

Westwood 

Danvers 

Lunenburg 

North  Adams 

Masconomet 

Gill  Montague 

Belchertown 

Needham 

Southbridge 

Edgartown 

Winthrop 

Quincy 

Foxborough 

Wachusett  Reg. 

Stoughton 

Reading 

Newburyport 

Northborough 

Triton 

Holbrook 


A 


EFFECTIVE 


Arrow  direction  Indicates  more  effective 
Noteworthy  Districts  in  italics 


Math  8  MCAS  1999  -  EFFECTIVE  DISTRICTS 


Harvard 

Newbury  port 

Tyngs  borough 

Stoneham 

Amherst  Pelham 

West  Bridgewater 

Westborough 

Lenox 

Ware 

Clinton 

Sandwich 

Chelsea 

North  Reading 

Hadley 

Quabbin 

Southern  Berkshire 

Sutton 

Medway 

Berlin  Boylston 

Lawrence 

Triton 

Newton 

Wachusett  Reg. 

Wayland 

Ludlow 

Gardner 

Easthampton 

Abington 

Berkshire  Hills 

Grafton 


MOST  EFFECTIVE 


Chatham 

Foxborough 

Stoughton 

Belchertown 

Acushnet 

Tantasqua 

Norwood 

Shirley 

Westford 

Acton  Boxborough 

Hampshire 

King  Philip 

Beverly 

Lexington 

Bedford 

West  Tisbury 

Carver 

Greenfield 

Mendon  Upton 

Northampton 

North  borough 

Granby 

Ipswich 

Pentucket  Regional 

Nantucket 

Frontier 

Methuen 

West  Springfield 

Walpole 

Pea body 


EFFECTIVE 


Arrow  direction  Indicates  more  effective 
Noteworthy  Districts  in  italics 
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Science  8  MCAS  1999  -  EFFECTIVE  DISTRICTS 


Stone  ham 

Westborough 

Tyngsborough 

Harvard 

North  Reading 

Hadley 

Monson 

Masconomet 

Amherst  Pelham 

Berlin  Boylston 

Bedford 

Pentucket  Regional 

Northampton 

Manchester 

Sandwich 

Lenox 

Medway 

Swampscott 

Wayland 

Pea body 

Southern  Berkshire 

Newburyport 

Quabbin 

Auburn 

West  Tisbury 

Easthampton 

Berkshire  Hilts 

Belchertown 

Gardner 

Lawrence 


MOST  EFFECTIVE 


West  Bridgewater 

Norwood 

Clinton 

Holbrook 

King  Philip 

Brookline 

Tantasqua 

North  Brookfield 

Leicester 

Dartmouth 

Lexington 

Ayer 

Longmeadow 

Woburn 

Triton 

Agawam 

Wachusett  Reg. 

Nantucket 

Hingham 

Concord 

Chelsea 

Berkley 

Belmont 

Medfield 

East  Longmeadow 

Braintree 

North  Andover 

Arlington 

Foxborough 

Hampden  Wilbraham 


EFFECTIVE 


Arrow  direction  Indicates  more  effective 
Noteworthy  Districts  in  italics 


Total  10  MCAS  1999  -  EFFECTIVE  DISTRICTS 


Lenox 
Harvard 
Shrewsbury 
Hadley 
Millbury 
Nauset 
Belmont 

Pentucket  Regional 

Westborough 

North  Adams 

Wachusett  Reg. 

Norton 

Lee 

Cohasset 

Swampscott 

Manchester 

Westwood 

Berkshire  Hills 

Littleton 

Carver 

Southbridge 

Brookline 

Sharon 

Ayer 

Chelsea 

Ashubrnham  Westminster 

Sandwich 

Fitchburg 

Belchertown 

Braintree 


MOST  EFFECTIVE 


Andover 
Ashland 
Stoneham 
West  Boylston 
Amherst  Pelham 
Evereff 
Ware 

Berlin  Boylston 
Georgetown 
Lowell 
Hampshire 
Medway 
Monson 
Gardner 
Old  Rochester 
Need  ham 
Arlington 
Dennis  Yarmouth 
Hudson 
Hanover 
Medfield 
Rockland 
Clinton 
Reading 
King  Philip 
Masconomet 
Auburn 
New  Bedford 
Acton  Boxborough  EFFECTIVE 
Lawrence 


Arrow  direction  indicates  more  effective 
Noteworthy  Districts  in  italics 


ELA  10  MCAS  1999  -  EFFECTIVE  DISTRICTS 


Lenox 

Shrewsbury  MOST 
Chelsea 

Littleton  a 

Stone  ham  ^ 

Cohasset 

Norton 

Millbury 

Belmont 

Nauset 

Carver 

Westwood 

Bralntree 

Pentucket  Regional 

Manchester 

Harvard 

Ayer 

Fitchburg 
North  Adams 
Rockland 
Sandwich 
Clinton 

Amherst  Pelham 
Sharon 

Berlin  Boylston 

Ashubrnham  Westminster 

Ashland 

Berkshire  Hills 

Hanover 

Brookline 


Southbridge 

Hopedale  A 
Hadley 

Westborough 

Dracut 

Swampscott 

Auburn 

Newburyport 

Wachusett  Reg. 

Dennis  Yarmouth 

Pioneer  Valley  Reg. 

Ware  ham 

Need  ham 

Bourne 

West  Boylston 

Oxford 

Ware 

Dedham 

Lawrence 

Hudson 

Everett 

Medway 

Old  Rochester 

Easton 

Central  Berkshire 
Haverhill 
Revere 
Arlington 

Falmouth  EFFECTIVE 
Hampshire 


Arrow  direction  Indicates  more  effective 
Noteworthy  Districts  in  italics 
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Math  10  MCAS  1999  -  EFFECTIVE  DISTRICTS 


Harvard 

Lenox 

Hadley 

Wachusett  Reg. 

Nauset 

Lee 

Shrewsbury 
Georgetown 
Pentucket  Regional 
Ayer 

Swampscott 

Belmont 

Southbridge 

Gardner 

Andover 

Berkshire  Hills 

Ware 

Hampshire 

Sharon 

Westborough 

Millbury 

Carver 

Cohasset 

Westwood 

North  Adams 

Monson 

Old  Rochester 

Lowell 

Belchertown 

Norton 


Brookline 

MOST  EFFECTIVE  Acton  Boxborough 

New  Bedford 


Manchester 

Ashubrnham  Westminster 
Everett 
Masconomet 
Holliston 
Reading 
Sandwich 
Dennis  Yarmouth 
Littleton 

Amherst  Pelham 
Ipswich 
Arlington 
Hudson 
Ashland 
Marlborough 
West  Boylston 
Fitchburg 
Medway 
Grafton 
Mohawk  Trail 
Northampton 
Needham 

Northboro  Southboro 
Auburn 
Ludlow 

Bedford  EFFECTIVE 
Hanover 


Arrow  direction  Indicates  more  effective 
Noteworthy  Districts  in  italics 


Science  10  MCAS  1999  -  EFFECTIVE  DISTRICTS 


Millbury 
Westborough 
Lenox 
Harvard 
Shrewsbury 
Medfield 
North  Adams 
Hadley 
Nauset 
Lee 

Webster 

Belmont 

Belchertown 

Brookline 

Norton 

Manchester 

Swampscott 

West  Boylston 

Wachusett  Reg. 

Pentucket  Regional 

Berkshire  Hills 

Ashubrnham  Westminster 

King  Philip 

Andover 

Clinton 

Fitchburg 

Chelsea 

Northbridge 

Westwood 

Quaboag  Regional 


MOST  EFFECTIVE 


Braintree 
Sandwich 
Medway 
Everett 

Berlin  Boylston 

Southbridge 

Dracut 

Cohasset 

Ralph  Mahar 

Ashland 

Lexington 

Sharon 

Littleton 

Watertown 

Holbrook 

Dighton  Rehoboth 

Lowell 

Carver 

Swansea 

Grafton 

Reading 

Monson 

Palmer 

North  Attleborough 

Concord  Carlisle 

Douglas 

Needham 

Arlington 

Amherst  Pelham 

Bellingham 


EFFECTIVE 


Arrow  direction  Indicates  more  effective 
Noteworthy  Districts  in  italics 


Total  MCAS  1999  -  EFFECTIVE  DISTRICTS 
All  3  Grades,  3  subjects 


Harvard 

Lenox 

Halifax 

Westborough 

Sandwich 

Pentucket  Regional 

Shrewsbury 

Brookline 

Chelsea 

Stone  ham 

Ayer 

Wachusett  Reg. 

North  Reading 

Belmont 

Arlington 

Berkshire  Hills 

Braintree 

Clinton 

Norwood 

Woburn 

Swampscott 

Northampton 

Tyngs  borough 

Newton 

Monson 

North  Adams 

Gardner 

Belchertown 

Lawrence 

Carver 


MOST  EFFECTIVE 
▲ 


Newburyport 

Sharon 

Everett 

Westwood 

Manchester 

Auburn 

Lee 

West  Bridgewater 
East  Longmeadow 
Quabbin 
Littleton 

Dennis  Yarmouth 

Millbury 

Harwich 

Lexington 

West  Boylston 

Chatham 

Bedford 

Medfield 

Franklin 

Uxbridge 

Dartmouth 

Falmouth 

Southbridge 

Medway 

Needham 

Central  Berkshire 

Norton 

Triton 

Plymouth 


Arrow  direction  Indicates  more  effective 
Noteworthy  Districts  in  italics 


APPENDIX  B:  Deriving  the  Effectiveness  Index 

The  Effectiveness  Index  (EI)  is  derived  by  comparing  actual  scores  on  standardized  tests 
with  scores  as  predicted  by  a  model  which  factors  in  the  role  commimity  characteristics 
play  in  educational  outcomes. 

The  Commimity  Effects  Factor  (CEF)  model,  the  statistical  basis  for  the  Effectiveness 
Index,  was  developed  in  a  doctoral  dissertation  {EducationAchievement  Communities:  A 
New  Model  for  ''Kind  of  Community"  in  Massachusetts  Based  on  an  Analysis  of  Community 
Characteristics  Affecting  Educational  Outcomes,  May  1998,  University  of  Massachusetts, 
Amherst).  That  work  is  the  basis  for  determining  school  effectiveness.  The  model 
examines  the  relationship  between  selected  demographic  characteristics  and  educational 
outcomes.  These  characteristics  include:  average  education  level,  average  income, 
poverty  rate,  single-parent  status,  language  spoken,  and  percentage  of  school-age 
population  enrolled  in  private  schools.  These  variables  were  chosen  because  they 
correlate  with  achievement  and  because  the  education  literature  identifies  them  as 
connected  to  academic  performance. 

In  order  to  refine  a  better  CEF  model,  it  is  first  necessary  to  factor  the  impact  of  these 
demographic  variables  on  each  other.  This  can  be  done  through  a  technique  known  as 
principal  component  analysis  that  is  a  statistical  mechanism  that  reduces  many  variables 
to  a  few  salient  ones  that  have  the  most  impact  on  an  outcome.  Once  the  factors  have 
been  identified,  a  regression  analysis  produces  the  equations  that  can  be  used  to  either 
build  a  kind-of-community  model  or  to  predict  expected  district 
performance  on  achievement  tests.  The  degree  to  which  a  commimity's  characteristics 
lifts  or  lowers  test  scores  is  reflected  in  a  Conmiunity  Effects  Factor  (CEF),  a  measure  of 
demography. 

The  CEF,  which  is  a  measure  of  the  demographic  lift  or  drag  of  each  commimity 
concerning  educational  achievement,  is  a  good  point  of  departure  for  analyzing  school 
and  school  district  effectiveness.  The  CEF  identifies  expected  levels  of  performance 
based  on  community  characteristics  which,  for  better  or  worse,  are  very  powerful 
indicators  of  educational  achievement  in  Massachusetts.  In  this  analysis,  Weston  is  the 
most  demographically  advantaged  community  in  the  state  in  terms  of  educational 
outcomes  (CEF  =  +  2.8),  and  Lawrence  is  the  least  advantaged  (CEF  =  -  4.8).  The  CEF 
has  a  strong  relationship,  or  correlation,  to  test  scores.  In  practical  application^  the  CEF 
establishes  the  methodology  for  predicting  likely  standardized  test  scores  based  on 
district  demography. 
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Correlation  is  a  process  that  identifies  the  interdependence  of  one  variable  with 
another.  Correlation  simply  shows  "the  extent  to  which  two  things  typically  run 
together."  [The  Economist,  6  Dec.  1997,  p.  82].  Correlation  is  not  equivalent  to  causation; 
it  can  only  reveal  tendencies  between  variables,  not  identify  caiises.  Correlations  simply 
demonstrate  relationships.  A  perfect  correlation  woxild  be  1.0.  For  example,  the 
correlation  between  inches  and  feet  is  1,0  because  it  is  a  perfect  linear  fit;  12  inches 
always  equals  one  foot.  Correlations  in  real  world  situations  involving  human 
behavior  are  never  1.0. 

The  dissertation  research  found  that  the  correlation,  or  the  connection,  between 
spending  (Per-Pupil  Expenditure  or  PPE)  and  achievement  in  Massachusetts  was  .28, 
which  is  relatively  low.  While  spending  dearly  matters,  merely  increasing  spending 
levels  has  a  relatively  weak  impact  on  results.  Increasingly,  many  people  are  coming  to 
the  realization  that  how  a  system  spends  money  is  more  important  than  how  much 
money  it  spends.  The  achievement  outcome  accounted  for  by  the  commimity  effects 
factor  (CEF)  is  much  stronger;  that  relationship  is  .85.  This  is  not  to  say  the  community 
context,  the  CEF,  is  the  most  important  determinant  of  school  success,  but  it  is  a 
significant  element  that  must  be  a  major  consideration  in  any  plan  to  improve 
education  in  disadvantaged  areas. 

The  Effectiveness  Index  was  generated  in  the  following  manner: 

o  Utilize  the  1999  MCAS  results  as  an  outcome  indicator  for  achievement  in  each  of  the 
state's  most  populous  200  communities.  (NOTE:  This  model  does  not  evaluate  results  in 
the  smallest  communities  of  the  state  which  comprise  about  3%  of  the  population.) 

o  Utilize  the  CEF  model  to  predict  a  score  for  each  district.  This  predicted  score  is 
based  solely  on  commimity  characteristic  as  they  affect  educational  outcomes. 

o  Compare  the  actual  to  the  predicted  score.  Systems  whose  actual  scores  are 
significantiy  higher  than  predicted  scores  and  whose  absolute  scores  are  at  or  above 
state  average  are  identified  as  effective.  Systems  with  positive  effectiveness  indexes  but 
scores  below  state  average  are  identified  as  noteworthy. 


For  more  information,  please  contact  the  UMass  Donahue 
Institute,  (617)  287-7055  or  contact  Robert  Gaudet, 
(617)  469-6843;  rgauHpt^wnrlH  c^td  rnm. 
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Statistical  Inf onnation 


Response:  TOTAL  MCAS  99 


Summary  of  Fit 

RSquare 

0.841836 

RSquare  Adj 

0.837019 

Root  Mean  Square  Error 

20.746 

Mean  of  Response 

2095.853 

Observations  (or  Sum  Wgts) 

204 

Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>ltl 

Std  Beta 

Intercept 

2090.7843 

3.684809 

567.41 

0.0000 

0 

F2ZC0L 

49.831473 

6.385676 

7.80 

<.0001 

0.368282 

F2Z  FAM$ 

26.815619 

6.020898 

4.45 

<.0001 

0.268135 

F2ZP0V 

-55.84579 

8.704618 

-6,42 

<.0001 

-0.37367 

F2ZPRVSCHL 

-7.447792 

3.505809 

-2.12 

0.0349 

-0.0665 

F22NONENG 

-5.506262 

2.511059 

-2.19 

0.0295 

-0.09849 

F2ZSPAR 

■ 

6.2973197 

6.14113 

1.03 

0.3064 

0.033331 

Effect  Test 


Source 

Nparm    DP    Sum  of  Squares 

F  Ratio 

Prob>F 

F2ZCOL 

1        1  26209.710 

60.8967 

<.0001 

F22  FAM$ 

1       1  8537.322 

19.8360 

<.0001 

F2ZPOV 

1       1  17715.362 

41.1606 

<.0001 

F2ZPRVSCHL 

1        1  1942.438 

4.5131 

0.0349 

F2ZNCX^NG 

1        1  2069.515 

4.8084 

0.0295 

F2ZSPAR 

1        1  452.568 

1.0515 

0.3064 

f  N 

Whole-Model  Test 


1900  1950  2000  2050  2100  2150  2200  2250 
TOTAL  MCAS  99  Predicted 


Analysis 

Of 

Variance 

Source 

DP 

Sum  of  Squares 

Mean  Square 

P  Ratio 

Model 

6 

451289.51 

75214.9 

174.7573 

Error 

197 

84788.08 

430.4 

Prob>P 

C  Total 

203 

536077.59 

<.0001 

